On Dialogue Breakdown: Annotation and Detection A report from the dialogue breakdown detection challenge

We organized the dialogue breakdown challenge, which is an evaluation workshop dedicated to the detection of breakdowns in Japanese human-machine chat-oriented dialogues. This paper reports analyses made on the breakdown annotation to the evaluation data used in the challenge and on behaviors of the detection systems submitted by six teams. The performances of the ensembles of the systems are also reported. Moreover, we investigate the effects of aggregating multiple models learned from different annotators. In response to the results of the investigation and in consideration of the highly subjective nature of breakdown, we propose ’not deciding unique correct labels’ in both training and evaluation of the systems.